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Method of Speech Recognition with 
Compensation for both Channel Distortion and Background Noise 

Field of Invention 

This invention relates to speech recognition and more particularly to compensation for 
5 both backgrovmd noise and channel distortion. 

Background of Invention 

. A speech recognizer trained with relatively a quiet office environment speech data and 

© operating in a mobile environment may fail due to at least to the tow distortion sources of back 
in groimd noise and microphone changes. The background noise may be from a computer fan, car 
^ engine, and/or road noise. The microphone changer may be due to the quality of the 

HI 

* microphone, hand-held or hands-free and, a position to the mouth. In mobile application of 
7" speech recognition, both the microphone conditioner and background noise are subject to 
6 change. 

S Cepstral Mean Normalization (CMN) removes utterance mean and is a simple and 

effective way of dealing with convolutive distortion such as telephone channel distortion. See 
"Effectiveness of Linear Prediction Characteristics of the Speech Wave for Automatic Speaker 

20 Identification and Verification" of B. Atal in Journal of Acoustics Society of America, Vol. 55: 
1304-1312, 1974. Spectral Subtraction (SS) reduces background noise in the feature space. See 
article "Suppression of Acoustic Noise in Speech Using Spectral Subtraction" of S.F. Boll in 
IEEE Transactions on Acoustics, Speech and Signal Processing, ASSP-27(2): 113429, April 
1979. Parallel Model Combination (PMC) gives an approximation of speech models in noisy 

25 condition from noise-free speech models and noise estimates. See "An Improved Approach to 
the Hidden Markov Model Decomposition of Speech and Noise" of MJ. F. Glaes and S. Yoxmg 
in Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, 
Volume 1, pages 233-236, U.S.A., April 1992. The techniques do not require any training data. 
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Joint compensation of additional noise and convolutive noise can be achieved by the 
introduction of a channel model and a noise model. A spectral bias for additive noise and a 
cepstral bias for convolutive noise are introduced in an article by M. Afify, Y. Gong, and J. P. 
Haton, This article is entitled "A General Joint Additive and Convolutive Bias Compensation 
5 Approach Applied to Noisy Lombard Speech Recognition" in IEEE Trans, on Speech and Audio 
Processing, 6(6): 524-538, November 1998. The five biases can be calculated by application of 
Expectation Maximization (EM) in both spectral and convolutive domains. A procedure by J.L. 
Gauvain, et al, is presented to calculate convolutive component, which requires rescanning of 
training data. See J.L. Gauvain, L. Lamel, M. Adda-Decker, and D. Matrouf entitled 

10 "Developments in Contmuous Speech Dictation using the ARPA NAB News Task." In Proc. of 
2 IEEE International Conference on Acoustics, Speech, and Signal Processing, pages 73-76, 
Q Detroit, 1996. Solution of the convolutive component by steepest descent method has also been 

11 reported. See Y. Minami and S. Furui entitled "A Maximum Likelihood Procedure for a 
^ Universal Adaptation Method Based on HMM Composition." See Proc. of IEEE Intemational 
S Conference on Acoustics, Speech and Signal Processing, pages 129-132, Detroit, 1995. A 
J-^ method by Y. Minami and S. Furui needs additional universal speech models, and redestination 
p of channel distortion with the universal models when channel changes. See Y. Minami and S. 
H| Furui entitled "Adaptation Method Based on HMM Composition and EM Algorithm" in 

Proceedings of IEEE Intemational Conference on Acoustics, Speech and Signal Processing, 
20 pages 327-330, Atlanta 1996. 

The techniques presented by M.F.J. Gales in "PMC for Speech Recognition in Additive 
and Convolutional Noise," Technical Report TR-154, CUED/F-INFENG, December 1993 needs 
two passes of the test utterance, e.g., parameter estimation followed by recognition, several 
25 transformations between cepstral and spectral domains, and a Gaussian mixture model for clean 
speech. 

Altematively, the nonlinear changes of both type of distortions can be approximated by 
linear equations, assuming that the changes are small Jacobian approach, which models speech 
30 model parameter changes as the product of a jacobian matrix and the difference in noisy 
conditions, and statistical linear approximation are along this direction. See S. Sagayama, Y. 
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Yamaguchi, and S. Takahashi entitled "Jacobian Adaptation of Noisy Speech Models," in 
Proceedings of IEEE Automatic Speech Recognition Workshop, pages 396-403, Santa Barbara, 
CA, USA, December 1997. IEEE Signal Processing Society. Also see "Statistical Linear 
Approximation for Environment Compensation" of N.S. Kim, IEEE Signal Processing Letters, 
5 5(1):8-10, January 1998. 

Maximum Likelihood Linear Regression (MLLR) transforms HMM parameters to match 
the distortion factors. See '^Maximum Likelihood Linear Regression for Speaker Adaptation of 
Continuous Density HMMs" by C.J. Leggetter and P.C. Woodland in Computer, Speech and 
10 Language, 9(2): 171-185, 1995. This method is effective for both sources but requires traming 
data and introduces the dependence to the speakers. 

Iff Summary of Invention 

g In accordance with one estabUshment of the present inventor a new method that Handles 

s simultaneously noise and channel distortions to make a speaker independent system robust to a 
wide variety of noises and channel distortions. 

O Description of Drawing 

% 

Figure 1 illustrates a speech recognizer according to one embodiment of the present 
invention; and 

Figure 2 illustrates the method of the present invention generating 

25 
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Description of Preferred Embodiments of the Present Inventions 



Referring to Fig. 1 there is illustrated a speech recognizer according to the present 
invention. The speech is appUed to recognizer 11. The speech is compared to Hidden Markov 

5 Models (HMM) 13 to recognize the text. The models initially provided on those with speech 
recorded in a quiet environment and the microphone of good quality. We want to develop a 
speech model set suitable for operating in the simultaneous presence of channel/microphone 
distortion and background noise. In accordance with the present invention, a speech model set is 
provided using statistics about the noise and speech. A low computation cost method integrates 

10 both PMC and CMN. 

O Referring to Figure 2, the first Step 1 is to start with HMM models trained on clean 

Ul 

U speech, with cepstral mean normahzation. We modify these models to get models to compensate 

m 

%1 for channel/microphone distortion (convolutive distortion) and simultaneous background noise 
S (additive distortion). For an HMM model, we have a lot of parameter but only change one 
D subset of the parameters and that is mean vectors m^j.^ The mean vectors mpj^k of the original 
l2 model space is modified where p is the index of the Probability Density Function (PDF), j is the 
ro state and k is the mixing component. 

20 The second Step 2 is to calculate which is the mean mel-scaled cesptrum coefficients 

(MFCC) vector over the trained database. Scan all data and calculate the mean to get b . 

The third Step 3 is to add mean b to each of this mean vector pool represented by rxipj^ k 
equation (1) to get: 

25 

mpik^mpj^k^h. (1) 



For example, there could be 100 PDFs, 3 states per PDF and 2 vectors per state, or a total of 600 
vectors. 

30 
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The fourth Step 4 is for a given input test utterance, an estimate of the background noise 
vector X is calculated. 

Let u^A [ux^u2^-uo\^ ^ ^[vl>V2'-vl)]^' where / means that the values are 
represented in log-spectral domain. 

We introduce the combination operator © such that: 

w'A«'eV=[^,^,...^;f (2) 

with 

wj = log(exp(2^'.) + exp(^'. )) (3) 

In Step 5, we calculate the mean vectors adapted to the noise X using equation 4. 

mpj^u = IDFTiDFT(mpj,,) © DFT(X )). (4) 
where DFT and IDFT are, respectively, the DFT and inverse DFT operation, mpj^k is the noise 
- compensated mean vector. 

Equation 4 involves several operators. DFT is Discrete Fourier Transform and IDFT is the 
Inverse Discrete Fourier Transform. The ® is an operation with two vectors. A © B = C. How 
® is defined, we look at equations 2 and 3. Equation 2 says the operation + operates on two 

vectors u and v snd the result is a vector of D dimension or [J^.J^'^'Wd^^ where T is the 
transposition. We take the two vectors and produce another vector. We need to specify each 
element in the resultant vector. Equation 3 says that the jth element in that vector ( ) is 
defined by the exponential of the element of u added to the exponential if the jth element of v 
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and take the log of the combination of the exponential of u added to the exponential of the j the 
element of v. This completes the definition of Equation 4. 

In the following steps, we need to remove the mem vector b of the noisy data y over the 
noisy speech space iAr(fi-om the resultant model). One may be able to synthesize enough noisy 
data from compensated models but this requires a lot of calculation. In accordance with the 
present invention the vector is calculated using statistics of the noisy models. The whole 
recognizer will operate with CMN (cepstral mean normalization mode), but the models in 
Equation 4 are no longer mean normahzed. We have dealt with additive noise. The second half 
of the processing is removing the cepstral mean of our models defined in Equation 4. This is not 
difficuh because we have the models in Equation 4. hi Step 6, we need to integrate aU the 
samples generated by Equation 4 to get the mean. Mean is b . Equation 5 is this integration. 

Let , be the variable denoting PDF index, Jbe the variable for state index, and Xhe the variable 
for mixing component index. 

b=E{y} (5) 



P j k 



Since 



P{y \pj.k) = ^{yJDFT{DFT{mpj^k) ® DFT(X )), ^p,-^) (6) 



We have 



^ = Z Z S P>(P)Pjl (f\ P)P%L3 {k\pJ)^pjM (7) 



P j k 
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Equation 7 shows that b can be worked out analytically, and it is not necessary to do the 
physically generation and integration. The final result is Equation 7 which is the integration into 
several sums. Sums over probability density functions and the sum over states and sum over 
mixing components. Then you have several quantities. The Py{ is the probabihty of having the 
PDF index. The given is the probability of being in the state if given the PDFp. The next is 
the probability the mixing component p J given we have the PDF index. The mean vector of the 

compensated mode. To make this complete we remove this b jfrom the compensated model to 
get the target model. This is Step 7. The target model is: 

'^pj.k^^pM-^ (8) 



This is what we want to load into our recognizer. This operation is done for each utterance. 

Calculation of b thus requires the knowledge of probability of each PDF. There 
are two issues with P 0 = p): 

• It needs additional storage space. 

• It is dependent of the recognition task e.g. vocabulary, grammar. 

Although it is possible to obtain that probability, we want to consider the following 
simplified cases. 

This operation to calculate the b can be simplified with three approximations. The first 
one uses equal probabilities for P;k{p) or constraint C. 

1 . Use equal probabilities for P^(j>): 

Pjc(p) = C (9) 
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2. Use equal probabilities for Pjc(p), Pj \P(j\p) and Px\„ 3(k\hJ). 



P«-(p) = C 

P3\H(j\p) = T> (10) 
Pv\H..j(k\pJ) = E 

3. In fact, the case described in Eq-10 consists in averaging the compensated 
mean vectors in^^,*. Referring to Eq-4 and Eq-1, it can be expected that the averaging reduces 
the speech part mpj,;t just as CMN does. Therefore, Eq-7 could be further simpUfied into: 

h =IDFT{DFT(\i)®DFT(X)). (11) 

The model m^^.^ , is then used with CMN on noisy speech. Unfortunately, b is a fimction of 
both channel and background noise in all above cases. In other words, in presence of noise, there 
is no guarantee that the channel will be removed by such a vector, as is for CMN. 

A subset of WAVES database containing hands-free recordings was used, which consists 
of three recording sessions: parked-tm (car parked, engine off), parked (car parked, engine off), 
and city-driving (car driven on a stop and go basis). 

In each session, 20 speakers (10 male) read 40 sentences each, giving 800 utterances. 
Each sentence is either 10, 7 or 4 digit sequence, with equal probabiUties. The database is 
sampled at 8kHz, with MFCC analysis frame rate of 20ms. Feature vector consists of 10 statis 
and 10 dynamic coefficients. 

HMMs used in all experiments are trained in TIDIGITS clean speech data. Utterance- 
based cepstral mean noimaUzation is used. The HMMs contain 1957 mean vectors, and 270 
diagonal variances. Evaluated on TIGIDIT test set, the recognizer gives 0.36% word error rate. 
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To improve the performance in noisy environment, the variances of the Gaussian PDFs 
can be MAP adapted to some slightly noisy data, e.g. WAVES parked_eval data. Such 
adaptation will not affect recognition of clean speech, but will reduce variance mismatch 
between HMMs the noisy speech.The new algorithm is referred to as JAC (joint compensation of 
additive noise and convolutive distortion). 



PARKED 



DRIVING 





BASELINE 


1.38 


30.3 




CMN 


0.59 


18.8 




PMC 


1.74 


6.29 




JAC FULL PROB (Eq-7) 


0.84 


1.93 




JAC CONST PDF PROB (Eq-9) 


0.84 


2.00 




JAC AVERAGE (Eq-10) 


1.16 


2.18 




JAC SIMPLIFIED (Eq-1 1) 


0.48 


1.91 



Table 1 : word error rate (WER) (%) as function of driving conditions and compensations 
If) methods 

Table-1 shows that: 



15 



Compared to noise-free recognition (WER) (0,36%), without any compensation 
(BASELENE) the recognition performance degrades severely. 



CMN effectively reduces the WER for parked data, but is not effective for driving 
conditions where additive noise becomes dominant. 



20 



• PMC substantially reduces the WER for driving conditions, but gives poor results for 
parked data where microphone mismatch is dominant. 



All JAC cases give lower WER than non-JAC methods. 
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Simplifying Eq-7 to Eq-9 then to Eq-10 results in progressive increase in WER, 
although the degradation is not severe. Especially, information in PDF probability is 
not critical to the performance. 

Simplijaed JAC gives lowest WER in all tests. For this hands-free speech recognition, 
the new met reduces word error rate by 61% for parked condition and 94% for city 
driving condition. 
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